Search

Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. We present COME-robot, the first closed-loop framework utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios. We meticulously construct a library of action primitives for robot exploration, navigation, and manipulation, serving as callable execution modules for GPT-4V in task planning. On top of these modules, GPT-4V serves as the brain that can accomplish multimodal reasoning, generate action policy with code, verify the task progress, and provide feedback for replanning. Such design enables COME-robot to (i) actively perceive the environments, (ii) perform situated reasoning, and (iii) recover from failures. Through comprehensive experiments involving 8 challenging real-world tabletop and manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~25%) compared to state-of-the-art baseline methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning.

Confronting the challenges of data scarcity and advanced motion synthesis in human-scene interaction modeling, we introduce the TRUMANS dataset alongside a novel HSI motion synthesis method. TRUMANS stands as the most comprehensive motion-captured HSI dataset currently available, encompassing over 15 hours of human interactions across 100 indoor scenes. It intricately captures whole-body human motions and part-level object dynamics, focusing on the realism of contact. This dataset is further scaled up by transforming physical environments into exact virtual models and applying extensive augmentations to appearance and motion for both humans and objects while maintaining interaction fidelity. Utilizing TRUMANS, we devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length, taking into account both scene context and intended actions. In experiments, our approach shows remarkable zero-shot generalizability on a range of 3D scene datasets (e.g., PROX, Replica, ScanNet, ScanNet++), producing motions that closely mimic original motion-captured sequences, as confirmed by quantitative experiments and human studies.

Point masses connected by springs, or mass-spring systems, are widely used in computer animation to approximate the behavior of deformable objects. One of the restrictions imposed by these models is that points that are not topologically constrained (linked by a spring) are unable to interact with each other explicitly. Such interactions would introduce a new dimension for artistic control and animation within the computer graphics community. Beyond graphics, such a model could be an effective proxy to use for model-based learning of complex physical systems such as molecular biology. We propose to imbue masses in a mass-spring system with electrostatic charge leading a system with internal forces between all pairs of charged points -- regardless of whether they are linked by a spring. We provide a practical and stable algorithm to simulate charged mass-spring systems over long time horizons. We demonstrate how these systems may be controlled via parameters such as guidance electric fields or external charges, thus presenting fresh opportunities for artistic authoring. Our method is especially appropriate for computer graphics applications due to its robustness at larger simulation time steps.

We give an algorithm that finds a zero forcing set which approximates the optimal size by a factor of $\text{pw}(G)+1$, where $\text{pw}(G)$ is the pathwidth of $G$. Starting from a path decomposition, the algorithm runs in $O(nm)$ time, where $n$ and $m$ are the order and size of the graph, respectively. As a corollary, we obtain a new upper bound on the zero forcing number in terms of the fort number and the pathwidth. The algorithm is based on a correspondence between zero forcing sets and forcing arc sets. This correspondence leads to a new bound on the zero forcing number in terms of vertex cuts, and to new, short proofs for known bounds on the zero forcing number.

We introduce a new tool useful for greedy coloring, which we call the forb-flex method, and apply it to odd coloring and proper conflict-free coloring of planar graphs. The odd chromatic number, denoted $\chi_{\mathsf{o}}(G)$, is the smallest number of colors needed to properly color $G$ such that every non-isolated vertex of $G$ has a color appearing an odd number of times in its neighborhood. The proper conflict-free chromatic number, denoted $\chi_{\mathsf{PCF}}(G)$, is the smallest number of colors needed to properly color $G$ such that every non-isolated vertex of $G$ has a color appearing uniquely in its neighborhood. Our new tool works by carefully counting the structures in the neighborhood of a vertex and determining if a neighbor of a vertex can be recolored at the end of a greedy coloring process to avoid conflicts. Combining this with the discharging method allows us to prove $\chi_{\mathsf{PCF}}(G) \leq 4$ for planar graphs of girth at least 11, and $\chi_{\mathsf{o}}(G) \leq 4$ for planar graphs of girth at least 10. These results improve upon the recent works of Cho, Choi, Kwon, and Park., Comment: 32 pages, 11 figures

Let ${\rm spex}(n,F)$ be the maximum spectral radius over all $F$-free graphs of order $n$, and ${\rm SPEX}(n,F)$ be the family of $F$-free graphs of order $n$ with spectral radius equal to ${\rm spex}(n,F)$. Given integers $n,k,p$ with $n>k>0$ and $0\leq p\leq \lfloor(n-k)/2\rfloor$, let $S_{n,k}^{p}$ be the graph obtained from $K_k\nabla(n-k)K_1$ by embedding $p$ independent edges within its independent set, where `$\nabla$' means the join product. For $n\geq\ell\geq 4$, let $G_{n,\ell}=S_{n,(\ell-2)/2}^{0}$ if $\ell$ is even, and $G_{n,\ell}=S_{n,(\ell-3)/2}^{1}$ if $\ell$ is odd. Cioab\u{a}, Desai and Tait [SIAM J. Discrete Math. 37 (3) (2023) 2228--2239] showed that for $\ell\geq 6$ and sufficiently large $n$, if $\rho(G)\geq \rho(G_{n,\ell})$, then $G$ contains all trees of order $\ell$ unless $G=G_{n,\ell}$. They further posed a problem to study ${\rm spex}(n,F)$ for various specific trees $F$. Fix a tree $F$ of order $\ell\geq 6$, let $A$ and $B$ be two partite sets of $F$ with $|A|\leq |B|$, and set $q=|A|-1$. We first show that any graph in ${\rm SPEX}(n,F)$ contains a spanning subgraph $K_{q,n-q}$ for $q\geq 1$ and sufficiently large $n$. Consequently, $\rho(K_{q,n-q})\leq {\rm spex}(n,F)\leq \rho(G_{n,\ell})$, we further respectively characterize all trees $F$ with these two equalities holding. Secondly, we characterize the spectral extremal graphs for some specific trees and provide asymptotic spectral extremal values of the remaining trees. In particular, we characterize the spectral extremal graphs for all spiders, surprisingly, the extremal graphs are not always the spanning subgraph of $G_{n,\ell}$.

Counts of small subgraphs, or graphlet counts, are widely applicable to measure graph similarity. Computing graphlet counts can be computationally expensive and may pose obstacles in network analysis. We study the role of cliques in graphlet counts as a method for graph similarity in social networks. Higher-order clustering coefficients and the Pivoter algorithm for exact clique counts are employed

In this work, we propose MetaScript, a novel Chinese content generation system designed to address the diminishing presence of personal handwriting styles in the digital representation of Chinese characters. Our approach harnesses the power of few-shot learning to generate Chinese characters that not only retain the individual's unique handwriting style but also maintain the efficiency of digital typing. Trained on a diverse dataset of handwritten styles, MetaScript is adept at producing high-quality stylistic imitations from minimal style references and standard fonts. Our work demonstrates a practical solution to the challenges of digital typography in preserving the personal touch in written communication, particularly in the context of Chinese script. Notably, our system has demonstrated superior performance in various evaluations, including recognition accuracy, inception score, and Frechet inception distance. At the same time, the training conditions of our model are easy to meet and facilitate generalization to real applications.

Data augmentation is a powerful technique to enhance the performance of a deep learning task but has received less attention in 3D deep learning. It is well known that when 3D shapes are sparsely represented with low point density, the performance of the downstream tasks drops significantly. This work explores test-time augmentation (TTA) for 3D point clouds. We are inspired by the recent revolution of learning implicit representation and point cloud upsampling, which can produce high-quality 3D surface reconstruction and proximity-to-surface, respectively. Our idea is to leverage the implicit field reconstruction or point cloud upsampling techniques as a systematic way to augment point cloud data. Mainly, we test both strategies by sampling points from the reconstructed results and using the sampled point cloud as test-time augmented data. We show that both strategies are effective in improving accuracy. We observed that point cloud upsampling for test-time augmentation can lead to more significant performance improvement on downstream tasks such as object classification and segmentation on the ModelNet40, ShapeNet, ScanObjectNN, and SemanticKITTI datasets, especially for sparse point clouds., Comment: This paper is accepted in 3DV 2024

The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn (by creating and collecting demonstrations) how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler that will optimize any DSPy pipeline to maximize a given metric. We conduct two case studies, showing that succinct DSPy programs can express and optimize sophisticated LM pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot prompting (generally by over 25% and 65%, respectively) and pipelines with expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top of that, DSPy programs compiled to open and relatively small LMs like 770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at https://github.com/stanfordnlp/dspy

High-resolution multi-modality information acquired by vision-based tactile sensors can support more dexterous manipulations for robot fingers. Optical flow is low-level information directly obtained by vision-based tactile sensors, which can be transformed into other modalities like force, geometry and depth. Current vision-tactile sensors employ optical flow methods from OpenCV to estimate the deformation of markers in gels. However, these methods need to be more precise for accurately measuring the displacement of markers during large elastic deformation of the gel, as this can significantly impact the accuracy of downstream tasks. This study proposes a self-supervised optical flow method based on deep learning to achieve high accuracy in displacement measurement for vision-based tactile sensors. The proposed method employs a coarse-to-fine strategy to handle large deformations by constructing a multi-scale feature pyramid from the input image. To better deal with the elastic deformation caused by the gel, the Helmholtz velocity decomposition constraint combined with the elastic deformation constraint are adopted to address the distortion rate and area change rate, respectively. A local flow fusion module is designed to smooth the optical flow, taking into account the prior knowledge of the blurred effect of gel deformation. We trained the proposed self-supervised network using an open-source dataset and compared it with traditional and deep learning-based optical flow methods. The results show that the proposed method achieved the highest displacement measurement accuracy, thereby demonstrating its potential for enabling more precise measurement of downstream tasks using vision-based tactile sensors.

Multimodal stock trading volume movement prediction with stock-related news is one of the fundamental problems in the financial area. Existing multimodal works that train models from scratch face the problem of lacking universal knowledge when modeling financial news. In addition, the models ability may be limited by the lack of domain-related knowledge due to insufficient data in the datasets. To handle this issue, we propose the Prompt-based MUltimodal Stock volumE prediction model (ProMUSE) to process text and time series modalities. We use pre-trained language models for better comprehension of financial news and adopt prompt learning methods to leverage their capability in universal knowledge to model textual information. Besides, simply fusing two modalities can cause harm to the unimodal representations. Thus, we propose a novel cross-modality contrastive alignment while reserving the unimodal heads beside the fusion head to mitigate this problem. Extensive experiments demonstrate that our proposed ProMUSE outperforms existing baselines. Comprehensive analyses further validate the effectiveness of our architecture compared to potential variants and learning mechanisms., Comment: 9 pages, 3 figures, 7 tables. Accepted by 2023 KDD Workshop on Machine Learning in Finance

With the development of quantum technology, hybrid devices that combine superconductors (S) and semiconductors (Sm) have attracted great attention due to the possibility of engineering structures that benefit from the integration of the properties of both materials. However, until now, none of the experiments have reported good control of band alignment at the interface, which determines the strength of S-Sm coupling and the proximitized superconducting gap. Here, we fabricate hybrid devices in a generic way with argon milling to modify the interface while maintaining its high quality. First, after the milling the atomically connected S-Sm interfaces appear, resulting in a large induced gap, as well as the ballistic transport revealed by the multiple Andreev reflections and quantized above-gap conductance plateaus. Second, by comparing transport measurement with Schr\"odinger-Poisson (SP) calculations, we demonstrate that argon milling is capable of varying the band bending strength in the semiconducting wire as the electrons tend to accumulate on the etched surface for longer milling time. Finally, we perform nonlocal measurements on advanced devices to demonstrate the coexistence and tunability of crossed Andreev reflection (CAR) and elastic co-tunneling (ECT) -- key ingredients for building the prototype setup for realization of Kitaev chain and quantum entanglement probing. Such a versatile method, compatible with the standard fabrication process and accompanied by the well-controlled modification of the interface, will definitely boost the creation of more sophisticated hybrid devices for exploring physics in solid-state systems., Comment: 18 pages, 9 figures

Shape transformation, a key mechanism for organismal survival and adaptation, has gained importance across fields as diverse as electronics and medicine. However, designing and controlling microscale shape-shifting materials remains a fundamental challenge in various actuation modalities. Here, we introduce SonoTransformer, an acoustically activated micromachine that delivers ultrafast transformability using preprogrammed soft hinges. These hinges concentrate energy through intensified oscillation and provide the necessary torque for transformation. We have created new machine designs to predetermine the folding state, enabling tailoring and milliseconds transformation. Additionally, we have shown selective transformation by adjusting acoustic power, realizing high degrees of control and functional versatility. Our findings open new research avenues in acoustics, physics, and soft matter, offering new design paradigms and development opportunities in robotics, metamaterials, adaptive optics, and microtechnology.

Recent advancements in large-scale text-to-image diffusion models have enabled many applications in image editing. However, none of these methods have been able to edit the layout of single existing images. To address this gap, we propose the first framework for layout editing of a single image while preserving its visual properties, thus allowing for continuous editing on a single image. Our approach is achieved through two key modules. First, to preserve the characteristics of multiple objects within an image, we disentangle the concepts of different objects and embed them into separate textual tokens using a novel method called masked textual inversion. Next, we propose a training-free optimization method to perform layout control for a pre-trained diffusion model, which allows us to regenerate images with learned concepts and align them with user-specified layouts. As the first framework to edit the layout of existing images, we demonstrate that our method is effective and outperforms other baselines that were modified to support this task. Our code will be freely available for public use upon acceptance.

Sequential models that encode user activity for next action prediction have become a popular design choice for building web-scale personalized recommendation systems. Traditional methods of sequential recommendation either utilize end-to-end learning on realtime user actions, or learn user representations separately in an offline batch-generated manner. This paper (1) presents Pinterest's ranking architecture for Homefeed, our personalized recommendation product and the largest engagement surface; (2) proposes TransAct, a sequential model that extracts users' short-term preferences from their realtime activities; (3) describes our hybrid approach to ranking, which combines end-to-end sequential modeling via TransAct with batch-generated user embeddings. The hybrid approach allows us to combine the advantages of responsiveness from learning directly on realtime user activity with the cost-effectiveness of batch user representations learned over a longer time period. We describe the results of ablation studies, the challenges we faced during productionization, and the outcome of an online A/B experiment, which validates the effectiveness of our hybrid ranking model. We further demonstrate the effectiveness of TransAct on other surfaces such as contextual recommendations and search. Our model has been deployed to production in Homefeed, Related Pins, Notifications, and Search at Pinterest., Comment: \c{opyright} {ACM} {2023}. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in KDD'23, http://dx.doi.org/10.1145/3580305.3599918

Pre-trained Language Models (PLMs) may be poisonous with backdoors or bias injected by the suspicious attacker during the fine-tuning process. A core challenge of purifying potentially poisonous PLMs is precisely finding poisonous dimensions. To settle this issue, we propose the Fine-purifying approach, which utilizes the diffusion theory to study the dynamic process of fine-tuning for finding potentially poisonous dimensions. According to the relationship between parameter drifts and Hessians of different dimensions, we can detect poisonous dimensions with abnormal dynamics, purify them by resetting them to clean pre-trained weights, and then fine-tune the purified weights on a small clean dataset. To the best of our knowledge, we are the first to study the dynamics guided by the diffusion theory for safety or defense purposes. Experimental results validate the effectiveness of Fine-purifying even with a small clean dataset., Comment: Accepted by Findings of ACL 2023

An efficient excited state method, named xCDFCI, in the configuration interaction framework, is proposed. xCDFCI extends the unconstrained nonconvex optimization problem in coordinate descent full configuration interaction~(CDFCI) to a multicolumn version, for low-lying excited states computation. The optimization problem is addressed via a tailored coordinate descent method. In each iteration, a determinant is selected based on an approximated gradient, and coefficients of all states associated with the selected determinant are updated. A deterministic compression is applied to limit memory usage. We test xCDFCI applied to H2O and N2 molecules under the cc-pVDZ basis set. For both systems, five low-lying excited states in the same symmetry sector are calculated together with the ground state. xCDFCI also produces accurate binding curves of carbon dimer in the cc-pVDZ basis with chemical accuracy, where the ground state and four excited states in the same symmetry sector are benchmarked.